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THE  DOD  TEST  AND  EVALUATION  PROCESS 


1.  Introduction. 

The  DoD  Test  and  Evaluation  (T&E)  Process  is  a  five  step  iterative  process  that 
provides  answers  to  critical  T&E  questions  for  decision  makers.  See  Figure  1~1 .  It 
establishes  a  comprehensive  approach  to  T&E  that  may  be  applied  one  or  more 
times  within  each  of  the  five  phases  of  the  DoD  Acquisition  Process  shown  in 
Figure  1  -2.  The  process  is  structured  so  that  it  can  be  used  at  any  time  by  any 
program. 
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Figure  1-1.  The  Five  Step  T&E  Process 

The  Milestone  0  decision  marks  the  beginning  of  the  acquisition  process  with 
its  approval  of  the  Mission  Need  Statement  (MNS).  The  MNS  defines  projected 
needs  in  broad  operational  terms.  The  Milestone  0  decision  also  identifies  the 
alternative  concepts  to  be  studied,  approves  the  start  of  Phase  0  %  Concept 
Exploration  and  Definition,  establishes  Phase  0  exit  criteria  to  be  evaluated  at 
Milestone  I,  and  activates  the  DoD  T&E  Process.  Milestone  decisions  are 
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documented  in  the  Acquisition  Decision  Memorandum  (ADM)  for  acquisition 
category  (AC AT)  I  programs. 

During  each 
phase  and  at  each 
Milestone,  the 
decision  maker 
needs  information 
from  several 
sources  to 
determine  the 
best  course  of 
action.  The 
information 
needed  from  T&E 
can  be 

summarized  as  a 
series  of 

questions  to  be  answered  during  each  phase.  The  questions  are  the  basis  for  the 
T&E  process  in  that  phase.  The  outcome  of  the  T&E  process  is  technical  and 
operational  assessments  delivered  to  the  decision  maker  as  answers  to  those 
questions.  The  questions  to  be  asked  in  each  phase  are  shown  below  in  paragraph 
3,  Application  to  the  Acquisition  Process. 

2.  DoD  T&E  Process. 

2.1.  T&E  Coordination  Function.  The  T&E  process  begins  with  the  T&E 
Coordination  Function,  which  has  three  parts:  the  Research,  Development  and 
Acquisition  (RDA)  Function  and  Steps  One  and  Five  of  the  Process. 

2.1.1.  Research,  Development,  and  Acquisition  (RDA)  Function  activities 
(Figure  2-1)  are  the  basis  for  T&E  activities  in  that  the  MNS  and  Operational 
Requirements  Document  (ORD)  identify  the  requirements  against  which  the  system 
must  be  evaluated.  Although  requirements  generation  and  maintenance  are  not 
part  of  the  T&E  process,  the  T&E  community  should  be  involved  to  ensure  that 
identified  requirements  are  testable,  measurable  and  can  be  evaluated.  Other  RDA 
activities  that  affect  T&E  range  from  specification  identification  through  design, 
development  and  production  to  training  and  support  needs.  The  degree  to  which 
each  of  these  has  been  demonstrated  must  be  considered  by  the  decision  maker. 
Answers  with  regard  to  the  achievement  of  system  performance  objectives  come 
from  T&E.  Answers  to  other  questions,  such  as  urgency,  military  effectiveness, 
costs  and  schedules,  come  from  other  sources  such  as  Intelligence,  Program 
Analysis  and  Evaluation  (PA&E),  Legal,  or  Comptroller. 
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At  Milestone  0  the  user  presents  the  MNS  to 
the  Defense  Acquisition  Board  (DAB).  If  the 
MNS  receives  favorable  consideration,  the  DAB 
authorizes  entry  into  Phase  0,  Concept 
Exploration  and  Definition,  from  which  are 
formulated  the  operational  suitability  and 
effectiveness  parameters  and  critical  technical 
parameters  (CTPs)  against  which  the  system  will 
be  tested  and  evaluated.  The  objectives  of 
Phase  0  are  to  explore  various  materiel 
alternatives  to  satisfy  the  documented  mission 
need,  define  the  most  promising  system  concept, 
develop  risk  analyses,  and  develop  a  proposed 
acquisition  strategy  and  initial  program  objective 
for  cost,  schedule  and  performance  for  the  most 
promising  system  concept(s).  A  Cost  and 
Operational  Effectiveness  Analysis  (COEA)  and 
ORD  are  prepared. 

As  the  acquisition  process  proceeds,  the 
concepts  to  be  studied  in  Phase  0  are  defined 
and  the  engineering  and  production  prototypes  to 
be  evaluated  in  Phases  I  and  II  are  built.  In 
Phases  III  and  IV,  the  system,  and  its 
modifications  if  appropriate,  must  continue  to  be 
evaluated. 

The  RDA  activities  list  in  Figure  2-1  includes  the  following  examples: 

2. 1.1.1.  Requirements  Definition.  System  requirements  evolve 
from  the  Mission  Needs  Statement  (MNS)  through  the  COEA  and  ORD  to  the 
Acquisition  Program  Baseline  (APB)  and  Test  and  Evaluation  Master  Plan  (TEMP). 
Requirements  must  be  clear,  complete,  consistent,  feasible,  and  evaluateable. 

They  are  derived  from  military  needs  and  stated  as  technical  and/or  operational 
suitability  and  effectiveness  parameters.  They  must  be  neither  wholly  qualitative, 
which  allows  uncontrolled  personal  opinion  to  enter  the  process,  nor  wholly 
quantitative,  forcing  "failure"  of  a  system  that  is  "good  enough"  but  not  perfect. 

Operational  requirements  are  a  basis  for  Critical  Technical  Parameters  (CTPs), 
used  by  the  developmental  T&E  (DT&E)  program  as  the  basis  for  its  tests. 
Operational  requirements  are  also  a  basis  for  Measures  of  Effectiveness  (MOEs) 
[e.g.,  missile  miss  distances,  kill  probabilities,  detection  and  tracking  ranges]  used 
by  the  operational  T&E  (OT&E)  agency  as  the  basis  for  its  determination  of 
operational  effectiveness  and  suitability.  Measures  of  Performance  (MOPs),  derived 
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from  MOEs,  may  also  be  used  to  support  the  determination  of  operational 
effectiveness  and  suitability.  MOEs  and  MOPs  form  the  basis  for  the  operational 
evaluation  which,  in  turn,  shapes  the  operational  tests. 

2.1. 1.2.  Digital  Modeling.  Digital  models,  implemented  on 
computers,  are  growing  in  importance,  use,  and  credibility.  Digital  models  used  in 
RDA  activities  may  be  similar  or  identical  to  those  used  in  the  T&E  Implementation 
Function,  and  in  fact  may  be  first  created  during  Concept  Exploration  and  Definition 
and  updated  and  used  as  the  system  proceeds  through  development  and  T&E. 
Models  are  used  during  the  RDA  activity  to  evaluate  concept  feasibility  and  to 
attempt  to  define  the  technical  limits  of  system  performance.  These  models  may 
be  simple  thought  processes  or  "back  of  the  envelope"  estimates,  or  they  may  be 
sophisticated  simulations  of  system  performance.  But  the  causal  relationship 
between  military  effectiveness  and  system  performance  must  always  be  developed, 
validated  and  documented. 

2. 1.1. 3.  Design.  System  design  is  the  first  step  in  creating  the 
system.  It  is  the  process  of  converting  specifications  into  a  visual  or  mathematical 
representation  of  the  system.  It  is  perhaps  the  most  critical  step  in  the  RDA 
function  because,  once  built,  the  system  is  much  more  difficult  to  modify  to  correct 
deficiencies  built  into  the  original  design.  Once  approved,  the  design  is  converted 
into  a  prototype.  Testing  is  conducted  to  assess  whether  system  design  meets  the 
requirements. 


2. 1.1. 4.  Prototyping.  Joint  Publication  1 -02  states  that  a 
prototype  is  a  model  suitable  for  evaluation  of  design,  performance,  and  production 
potential.  For  the  purpose  of  the  T&E  process,  prototypes  will  be  classified  as 
engineering  and  production. 

2. 1.1. 4.1.  Engineering  Prototype  (EP).  A  development  model 
of  a  unit  that  is  close  to  production.  This  may  apply  to  circuitry,  a  device,  or  a 
system,  and  may  be  in  a  breadboard  (technical)  configuration.  EPs  are  normally 
used  in  Phase  I. 


2. 1.1. 4.2.  Production  Prototypes  (PP).  A  final  model  of  a 
design  before  the  pilot  unit  is  approved  for  production.  It  should  be  highly 
representative  of  final  equipment,  except  that  the  exact  manufacturing  assembly 
process  and  production  design  changes  may  not  yet  be  used  or  incorporated.  It  is 
suitable  for  complete  evaluation  of  its  electrical  and/or  mechanical  form  and  may  be 
in  a  brassboard  (technical  and  operational)  configuration.  PPs  are  normally  used  in 
Phase  II. 


2. 1.1. 5.  Specifications  are  the  values  that  convert  requirements 
into  design  terms.  They  must  clearly  and  accurately  relate  back  to  the  technical 
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requirements  of  the  system.  An  audit  trail  from  the  requirements  documents  to  the 
specifications  ensures  that  the  system,  when  built  to  spec,  will  perform  as 
intended. 


2. 1.1. 6.  Reliability  Growth  is  a  process  in  which  the  reliability  of 
the  operational  system  improves  through  identification  and  correction  of  systemic 
reliability  failures.  During  Engineering  &  Manufacturing  Development  (EMD), 
laboratory  tests  on  prototypes  are  performed  to  uncover  component  reliability 
failures,  make  design  improvements,  and  project  a  reliability  point  estimate.  Later, 
the  system  is  deployed  and  field  data  are  collected.  The  validity  of  the  process  lies 
in  the  activities  associated  with  operational  failures.  If  the  user  is  aggressive  in 
reporting,  cataloging  and  investigating  reliability  failures,  and  commits  the  time  and 
resources  to  their  correction,  reliability  will,  in  all  likelihood,  improve.  If  these  steps 
are  not  taken,  reliability  cannot  improve  and  will  probably  decrease  as  the  system  is 
subjected  to  more  maintenance  than  was  originally  intended. 

2.1. 1.7.  Production  is  the  manufacturing  of  the  system  after  it  has 
been  funded,  found  to  be  producible,  and  deemed  operationally  suitable  and 
effective.  The  capability  and  integrity  of  the  manufacturing  process  will  be 
evaluated  compared  to  the  system  design  requirements  and  measured  through 
inspections  and  testing. 

2.1. 1.8.  Support  Equipment  is  that  category  of  ancillary  "things" 
necessary  to  sustain  the  system  or,  during  T&E,  to  support,  monitor  and  record  the 
test.  Support  equipment  includes  common  hand  and  power  tools,  cameras, 
vehicles,  and  a  myriad  of  other  items  without  which  the  system  could  not  operate 
properly.  Support  equipment  requirements  for  T&E  must  be  identified  early  in  the 
T&E  process  to  ensure  availability  when  needed  and  must  be  identified  in  Part  V  of 
the  TEMP. 

2.1. 1.9.  Training  Equipment.  Since  most  new  or  highly  modified 
systems  are  somewhat  unique,  the  personnel  who  will  use  and  maintain  the  system 
must  be  retrained  to  use  it  properly  and  safely.  Training  equipment  includes  those 
items  that  either  permit  that  training  or  are  used  during  training  to  instruct  the 
operators  and  maintainers  in  the  correct  way  to  operate  and  maintain  the  system. 

2.1.1.10.  Operational  Concept.  The  operational  concept,  as 
defined  in  the  ORD,  is  the  planned  methodology  by  which  the  system  will  be  used 
and  supported.  The  ORD  provides  performance  parameters  in  terms  of  operational 
suitability  and  effectiveness  criteria  thresholds  and  objectives.  Understanding  the 
operational  concept  is  critical  for  both  DT&E  and  OT&E  personnel. 

2.1.1.11.  Logistics  Support  Concept  is  the  plan  by  which  the 
system  will  be  maintained.  It  includes  spares,  transportation,  maintenance  and 
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support  personnel  requirements.  Developing  integrated  logistics  support 
requirements  consistent  with  readiness  objectives,  design  and  resources  should  be 
considered  early  in  the  acquisition  process. 

2.1.2.  STEP  ONE  is  the  identification  of  T&E  information  required  by  the 
Milestone  decision  maker.  See  Figure  2-2.  The  required  information  consists  of 
performance  and  effectiveness  evaluations  of  how  well  the  system  meets  the 
user's  needs.  This  information  progresses  from  answers  concerning  proposed 
alternative  concepts 
in  Phase  0  to 
answers  on  system 
technical 
performance  and 
operational  suitability 
and  effectiveness 
during  Phase  II  and 
beyond.  The  required 
information  usually 
centers  on  the 
current  system  under 
test  (SUT)  which  will 
be  in  the  form  of 
concepts,  prototypes, 
both  engineering  and 
production,  and/or 
the  system  itself, 
depending  on  the 
acquisition  phase. 

Step  One  of  the 
process  is  initiated  by 
the  preceding 

Milestone  decision  of  the  Acquisition  Process.  Areas  in  which  questions  can  be 
expected  from  the  Milestone  decision  maker  in  each  phase  are:  the  relevance  of 
historical  data,  accuracy  of  stated  requirements,  adequacy  of  T&E  infrastructure 
and  technology  base,  testing  alternatives,  system  performance  versus  validated 
threats,  projected  impact  of  the  system  on  battle  outcomes,  capabilities, 
limitations,  feasibility,  preferred  system,  and  T&E  exit  criteria.  Association  of  each 
question  with  an  acquisition  Phase  is  discussed  later  in  the  application  of  the 
Process  to  the  acquisition  Process.  The  principal  outcome  of  this  Step  is  the 
determination  of  evaluation  objectives. 

2.1.3.  STEP  FIVE  is  where  the  decision  maker  weighs  the  T&E  information 
against  other  programmatic  information  to  decide  a  proper  course  of  action.  See 
Figure  2-3.  The  decision  will  be  based  on  criteria  of  military  worth,  as  well  as 
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consideration  of  cost,  funding,  urgency,  etc.  When  associated  with  an  acquisition 
milestone,  this  decision  is  announced  in  an  ADM  which  outlines  the  future  course 
of  action  for  the  program  and  the  SUT.  Additionally,  each  milestone  decision  will 
contain  the  "exit  criteria"  for  the  next  Phase/Milestone. 


2.1.4.  Feedback.  It  is 
possible  that  the  decision 
maker  asked  the  wrong 
questions  or  that  the 
questions  were 
misunderstood  by  the  T&E 
community.  The  decision 
maker  should  compare  the 
information  contained  in  the 
assessments  with  the 
questions  previously  asked 
to  ensure  that  the  responses 
are  adequate.  Problems 
should  be  highlighted  and 
resolved.  The  decision 
maker  can  then  rephrase 
succeeding  questions  to 
ensure  better  understanding 
of  the  information  needed. 

2.2.  T&E  Implementation  Function.  The  test  and  evaluation  implementation 
function  encompasses  the  three  steps  necessary  to  develop  the  information  needed 
to  prepare  the  assessments  used  by  the  decision  maker  in  Step  Five. 

2.2.1.  STEP  TWO  is  the  pre-test  analysis  of  the  evaluation  objectives  from 
Step  One  to  determine  the  types  and  quantities  of  data  needed,  the  results 
expected  or  anticipated  from  the  tests,  and  the  analytical  tools  needed  to  conduct 
the  tests  and  evaluations.  See  Figure  2-4. 

2.2.1 .1 .  Pre-test  analysis  develops  the  analytical  tools,  allocates  test 
parameters  to  requirements,  estimates  test  results,  determines  the  types  and 
quantities  of  data  needed,  and  identifies  the  major  test  objectives. 

2.2.1 .2.  The  use  of  validated  models  and  simulation  systems  during 
pre-test  analysis  can  aid  in  determining:  how  to  design  test  scenario(s),  how  to  set¬ 
up  the  test  environment,  how  to  properly  instrument  the  test,  how  to  man  and 
control  the  test  resources,  how  best  to  sequence  the  test  trials,  and  how  to 
estimate  outcomes.  In  this  step,  models  and  simulations  are  used  to  estimate  test 
results. 


Figure  2-3.  Step  Five:  The  Decision  Maker  Weighs 
T&E  information  Against  Other  Program  information 
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2.2.1. 3. 
The  outcome  of  this 
step  is  the  expected 
outcome  of  the 
system  under  test. 
When  a 

determination  is 
made  that  additional 
data  are  necessary 
and  major  test 
objectives  are 
identified,  the 
process  moves  to 
Step  Three. 


Figure  2-4.  Step  Two:  Pre-Test  Analysis 

2.2.2.  STEP 
THREE,  test  activity 
and  data 

management ,  shown 
in  Figure  2-5,  is  the 
actual  test  activity 
planning,  test  conduct 
and  data 

management.  Given 
the  data  requirements 
from  Step  Two,  T&E 
managers  determine 
what  valid  data  exist 
in  historical  files  that 
can  be  applied  to  the 
SUT  and  what  new 
data  must  be 
developed  from  test 
events.  They  plan 
and  execute  the  tests 
necessary  to  develop  the  data.  The  historical  and  developed  data  are  reviewed  for 
completeness  and  accuracy,  authenticated,  and  forwarded  to  Step  Four  for 
assessment  as  measured  outcomes. 


Management 
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2.2.2. 1 .  Test  activity  planning  includes  gathering  the  required  test 
articles  and  test  support  equipment,  scheduling  facilities  and  identifying  the 
climatic,  mission  and  threat  environments. 

2. 2. 2. 2.  Test  conduct  is  the  culmination  of  test  activity  planning  and 
with  the  exception  of  historical  data,  is  the  execution  of  the  actual  test  events.  As 
shown  in  Figure  2-5.,  test  activities  range  from  historical  searches  of  the 
performance  of  like  or  similar  components,  subsystems  and  systems  to  actual 
hardware  tests  of  component  incoming  parts  through  multi-system,  open-air, 
operationally  realistic  "free-play"  scenarios.  Data  gathered  during  the  test,  or  from 
historical  searches,  are  input  to  the  data  management  activity  for  processing. 

2.2.2.2.1.  Historical  Test  Data.  The  initial  step  in  any  test  activity 
should  be  the  examination  of  previous  test  data  stored  in  historical  archives. 
Component,  subsystem  and  system  data  from  like  or  similar  components, 
subsystems  or  systems  must  be  examined  first  for  applicability,  and,  if  applicable, 
to  reduce  the  amount  and/or  type  of  hardware/software  testing  necessary  for 
evaluation.  Further,  modeling  and  simulation  of  historical  test  data,  in  lieu  of  actual 
testing  of  the  current  SUT,  should  be  performed  to  fill  historical  information  voids. 
This  is  a  highly  cost  effective  procedure  that  leads  to  the  determination  of  what 
data  are  lacking  and  therefore  needed  from  new  test  and  evaluation  events. 

2.2.2.2.2.  Component  Measurement  Test  Events  often  involve  the  use 
of  specialized  capabilities  to  explore  and  evaluate  advanced  technologies  and  are 
usually  the  first  test  events  performed  during  the  development  and/or  buildup  of 
the  system.  Examples  include  incoming  parts  inspection,  thermal,  acoustic  and 
vibration  cycling,  power  requirement  and  heat  generation  tests. 

2.2.2.2.3.  Integration  Test  Events  test  components,  subsystems  and 
systems  combined  with  other  elements.  The  other  elements  may  be  other  parts  of 
the  same  system  or  other  systems  with  which  the  SUT  must  operate.  These  tests 
are  frequently  conducted  in  integration  laboratories  specifically  designed  to  test  the 
SUT  integrated  with  other  systems  or  functions.  Integration  laboratories  are 
generally  weapon  system  specific  and  are  used  from  the  beginning  of  a  system's 
development  through  integration  and  fielding.  These  tests  employ  a  variety  of 
models,  simulations,  and  stimulations  to  generate  scenarios  and  background  at  or 
near  real  time. 

2.2.2.2.4.  Hardware-in-the-loop  {HITL)  Events  use  elements  of  the  SUT 
in  combination  with  software  to  examine  the  performance  of  those  elements  before 
the  entire  system  is  available  or  when  a  specific  capability  cannot  be  tested.  HITL 
events,  such  as  breadboard,  brassboard,  or  prototype  tests  permit 
system/subsystem  evaluation  during  various  stages  of  development. 


2. 2. 2. 2. 5.  Installed  Systems  Events  provide  capabilities  to  evaluate 
SUTs  that  are  installed  on  and  integrated  with  their  host  platforms.  These  tests 
can  occur  in  indoor  facilities  such  as  electronic  warfare  (EW)  or  climatic  chambers 
or  as  outdoor  DT  and  OT  tests.  Chambers  provide  a  secure  site  to  evaluate  the 
capabilities  and  limitations  of  the  system  against  simulated  and  stimulated  inputs. 
Climatic  chambers  examine  SUT  capabilities  in  varied  temperature  and  humidity 
conditions  without  having  to  transport  the  SUT  to  those  naturally  occurring 
climates. 


2.2.2.2.6.  Field  or  Open-air  Test  Events  refers  to  any  test  conducted  in 
an  open  environment.  It  includes  surface  (land  and  sea),  undersea,  airborne  and 
spaceborne  testing.  Field  tests  are  conducted  where  it  is  feasible,  safe  and  secure 
to  test  all  or  part  of  the  SUT  in  an  environment  that  is  normally  more  realistic  than 
any  attainable  indoors.  Field  tests  may  allow  the  SUT  to  be  operated  more  closely 
to  its  operational  conditions.  However,  particularly  with  EW  systems,  field  tests 
may  provide  less  insight  into  the  performance  of  a  system  because  indoor  facilities 
are  the  only  place  high  density,  high  fidelity  threat  signals  can  be  generated  in  a 
secure  environment. 

2.2.2.2.7.  Simulation/Stimulation  Events  are  used  extensively  in  the 
DoD  test  process.  They  can  be  applied  to  computer  or  physical  working  models  or 
the  SUT.  They  may  be  real  time  or  non-real  time  models.  Effective  use  of  credible 
models  and  their  simulation/stimulation  events  will  provide  cost  effective  T&E. 

2. 2. 2. 3.  Data  management.  Data  recorded  during  test  events  are  often 
not  in  a  form  best  suited  for  analysis.  Several  steps  are  taken  to  make  the  data 
more  usable: 

2.2.2.3.1.  Data  Collection  and  Reduction.  Most  data  are  recorded 
"raw",  scaled  to  match  the  recording  capabilities  of  the  analog  or  digital  recording 
system  without  regard  to  the  actual  magnitude  of  the  data.  This  first  step  in  the 
data  management  process  is  bringing  data  from  all  sources  together  and  reducing 
them  to  engineering  values. 

2. 2. 2. 3. 2.  Analysis  and  authentication  ensures  that  all  data  accurately 
reflect  the  operation  of  the  SUT.  Data  from  multiple  sources  are  compared  for 
agreement,  data  dropouts  are  filled  in  where  possible  and  questionable  data  are 
compared  with  other  sources  for  reasonableness. 

2. 2. 2. 3. 3.  The  data  are  then  distributed  for  further  use  in  the  synthesis 
and  evaluation  step. 

2.2.2.4.  Deficiency  Reporting  is  the  process  of  formally  documenting 
failures  to  meet  required  performance  thresholds  or  objectives,  human  factors 
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limitations,  safety  concerns,  etc.  Deficiency  reports  are  forwarded  to  the  program 
office  for  correction.  Procedures  for  deficiency  reporting  differ  among  the  services, 
but  because  of  the  critical  importance  of  deficiency  reporting  during  field  testing,  it 
should  in  all  cases  be  a  clearly  defined,  formal  process.  Evaluation  of  the  impact  of 
the  deficiency  on  suitability  and  effectiveness  must  be  part  of  the  deficiency 
reporting  process.  Merely  quoting  specifications  as  a  justification  for  submitting  a 
deficiency  report  may  lead  to  costly,  time-consuming  changes  that  are  not 
operationally  required  or  that  preclude  making  other,  more  valuable  changes.  For 
example,  the  evaluators  should  assess  how  the  deficiency  affects  operational 
mission  accomplishment.  Does  the  deficiency  require  more  spares,  more 
manpower  or  longer  down  times  than  can  be  afforded  in  a  military  operation?  And 
finally,  can  the  user  work  around  these  limitations  and  make  effective  use  of  the 
system? 

2. 2. 2. 5.  The  end  product  of  this  step  is  measured  outcomes  in  the  form  of 
test  and  evaluation  activity  reports  which  are  provided  to  the  analysts  for  Step 
Four. 

2.2.3.  STEP 
FOUR,  post  test 
synthesis  and 
evaluation,  is  the 
combination  of  the 
measured  outcomes 
of  Step  Three  with 
the  expected 
outcomes  from 
Step  Two, 
tempered  with 
technical  and 
operational 
judgement.  The 
output  of  Step  Four 
is  the  answers  to 
the  questions 
developed  at  Step 
One.  See  Figure  2- 
6. 

When  measured  outcomes  differ  from  expected  outcomes,  the  test  conditions 
and  procedures  must  be  reexamined  to  determine  if  the  deviations  are  real,  that  is, 
due  to  unexpected  performance  of  the  SUT,  or  are  caused  by  test  limitations  such 
as  a  lack  of  fidelity  in  computer  simulation,  non-availability  of  support  assets,  or 
less  than  full  system  availability.  If  the  differences  are  due  to  test  limitations,  the 


Figure  2-6.  Step  Four:  Post-Test  Synthesis  and 


Evaluation 
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effect  of  the  limitations  must  be  evaluated,  if  possible,  and  judgement  used  to 
estimate  true  system  performance.  However,  since  this  may  involve  extrapolation 
of  test  data,  it  is  inherently  risky.  Despite  the  additional  cost  and  time,  retesting  is 
usually  prudent. 

In  this  step,  models  and  simulations  are  normally  used  to  process  test  data  and 
to  evaluate  system  performace  and  effectiveness  using  data  obtained  from  the 
tests.  The  assumptions  of  tactics,  environment,  system  performance  and  support 
must  be  carefully  chosen  and  fully  described  and  documented. 

2.2.4  Feed-  back.  As  with  the  decision  maker,  feedback  is  necessary  within 
the  T&E  process  to  ensure  the  quality  of  the  output.  But  unlike  the  decision  maker, 
who  needs  to  know  whether  the  questions  have  been  answered  satisfactorily,  here 
the  evaluators  must  ensure  that  the  data  are  sufficient  to  answer  the  questions 
posed  by  (or  as  understood  from)  the  decision  maker.  Their  satisfaction  with  the 
test  report  data  must  be  transmitted  as  feedback  to  the  persons  responsible  for  the 
pre-test  analysis  to  ensure  that  both  the  current  test  and  the  process  itself  continue 
to  be  as  complete,  effective  and  efficient  as  possible. 

This  step  concludes  with  the  preparation  of  technical  and  operational 
assessments  that  answer  the  decision  maker's  questions  at  Step  One. 

2.2.5.  The  T&E  process  concludes  at  Step  Five,  as  noted  above,  where  the 
decision  maker  weighs  the  T&E  information  against  other  program  information  and 
assesses  the  progress  of  the  SUT. 

3.  Application  to  the  Acquisition  Process. 

The  DoD  T&E  Process  can  be  applied  to  the  five  phases  of  the  Acquisition 
Process  shown  in  Figure  1-2.  Its  application  and  execution  are  based  on 
addressing  specific  questions  in  each  phase. 

3.1.  Milestone  0-Concept  Studies  Approval.  The  Milestone  (MS)  0 
decision  designates  alternative  concepts  for  study.  T&E  concerns  at  MS  0  are  data 
that  can  be  developed  during  Phase  0  that  support  a  MS  I  decision  with  respect  to 
the  concepts  and  the  development  of  recommended  T&E  "exit  criteria"  for  the 
Milestone  I  ADM. 

3.2.  Phase  0-Concept  Exploration  and  Definition.  This  phase  will  explore 
and  define  each  concept.  Both  technical  and  operational  assessments  will  be 
prepared.  As  a  minimum,  T&E  will  address  the  questions  in  Figure  3-1  for  each 
concept  or  concept  comparison. 
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Phase  0-Concept  Exploration  &  Definition 


For  each  concept, 

{ 1 )  What  T&E  data  exist?  Does  analysis  conclude  that  the  concept(s)  will  work? 
What  is  the  confidence  level  in  this  assessment? 

(2)  Can  requirements,  as  stated,  be  evaluated? 

(3)  Does  the  existing  T&E  infrastructure/technology  base  permit  evaluation? 
Approximately  what  is  the  cost  and  time  frame?  If  not,  approximately  what  is 
the  cost  and  time  frame  to  create  the  infrastructure/technology  base? 

(4)  What  are  concept(s)  capabilities/limitations  versus  threats? 

(5)  How  can  T&E  favorably  impact  risk  management  or  risk  reduction? 

(6)  What  alternatives  exist  to  testing/test  assets?  What  are  the  risks? 

(7)  Has  the  preliminary  TEMP  been  approved?  Are  the  critical  technical 
parameters,  the  minimum  acceptable  operational  requirements,  and  the 
critical  system  characteristics  included? 

(8)  Is  each  proposed  concept  feasible?  Is  one  concept  preferred? 

(9)  What  are  the  recommended  Phase  I  T&E  "exit  criteria"? 

(10)  Does  the  Early  Operational  Assessment  (EOA)  address  the  availability  of  and 
planning  for  resources  projected  for  OT&E  test  events. 


Figure  3-1 .  Phase  0  T&E  Questions 

3.3,  Milestone  1-Concept  Demonstration  Approval.  The  MS  I  decision, 
rendered  in  an  ADM,  selects  the  preferred  concept(s)  to  continue  development. 
This  concept(s)  will  evolve  into  EPs  for  continued  T&E.  The  T&E  outcome  will  be 
the  assessment  of  each  EP. 

3.4.  Phase  1-Demonstration  and  Validation.  This  phase  will  examine  each 
EP.  It  will  result  in  both  technical  and  early  operational  assessments.  As  a 
minimum,  T&E  will  address  the  questions  in  Figure  3-2  for  each  EP.  These 
questions  are  basically  identical  to  the  Phase  0  questions.  However,  because  the 
program  is  more  advanced,  more  data  at  higher  confidence  will  exist 
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Phase  (-Demonstration  &  Validation 

For  each  engineering  prototype  (EP), 

(1)  Existing  data  analysis  -  will  the  EP(s)  work?  What  is  the  confidence  level  in  this 
assessment? 

(2)  Can  the  requirements,  as  stated,  be  evaluated? 

(3)  Does  the  existing  T&E  infrastructure/technology  base  permit  evaluation? 
Approximately  what  is  the  cost  and  time  frame?  If  not,  approximately  what  is 
the  cost  and  time  frame  to  create  the  infrastructure/technology  base? 

(4)  What  are  the  capabilities/limitations  of  each  EP  versus  threats? 

(5)  How  can  T&E  favorably  impact  risk  management  or  risk  reduction? 

(6)  What  alternatives  exist  to  testing/test  assets?  What  are  the  risks? 

(7)  Has  TEMP  been  approved?  The  TEMP  must  contain  the  performance 
parameters  reflected  in  the  ORD,  COEA,  and  APB.  They  must  be  consistent. 

(8)  Is  each  proposed  EP  feasible?  Is  one  preferred? 

(9)  What  are  the  recommended  Phase  II  T&E  "exit  criteria"? 

( 10)  What  are  the  recommended  criteria  for  certification  of  readiness  for  final 
OT&E? 

(11)  Does  the  EOA  address  the  early  projection  of  potential  operational 
effectiveness  and  suitability  criteria? 


Figure  3-2.  Phase  (  T&E  Questions 


3.5.  Milestone  ll-Developmental  Approval.  The  MS  II  decision  will 
determine  the  continued  progress  of  the  selected  prototype(s)  for  continued 
development  as  production  prototypes  (PPs).  The  T&E  outcome  will  be  the 
assessment  of  each  selected  prototype. 


3.6.  Phase  11-Engineering  and  Manufacturing  Development.  This  phase  will 
examine  each  PP,  executing  Steps  Two,  Three,  and  Four  of  the  T&E  process.  It 
will  result  in  both  technical  and  operational  assessments.  As  a  minimum,  T&E  will 
address  the  issues  in  Figure  3-3  for  each  PP  or  for  comparison. 
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Phase  ll-Engineering  &  Manufacturing  Development 

For  each  production  prototype  (PP), 

(1)  For  DT&E,  what  are  the  capabilities  and  limitations  of  each  PP  being 
developed?  What  is  the  confidence  level  in  this  data/assessment? 

(2)  For  OT&E,  are  the  PPs  suitable  and  effective  in  satisfying  the  mission  need? 
What  is  the  confidence  level  of  this  assessment? 

(3)  Have  key  performance  objectives/thresholds  been  validated  versus  advanced 
threats? 

(4)  Are  the  PPs  feasible?  Do  they  satisfy  the  need?  Is  one  preferred? 

(5)  Have  requirements  changes  been  incorporated  into  the  APB,  ORD  and 
contract  specifications? 

(6)  Have  specification  changes  been  reflected  back  to  requirements  and 
incorporated  into  the  APB  and  ORD? 

(7)  Does  the  TEMP  reflect  the  changes  in  (5)  and  (6)?  Has  it  been  approved? 

(8)  Should  there  be  a  Milestone  IV,  what  are  the  recommended  Phase  III  T&E  "exit 
criteria"? 


Figure  3-3.  Phase  II  T&E  Questions 

3.7.  Milestone  Ill-Production  Approval.  The  MS  III  decision  will  determine 
whether  to  enter  production  and  deploy  one  or  more  of  the  systems  under 
development. 


3.8.  Phase  Ill-Production  and  Deployment.  This  phase  will  examine  each 
selected  system  executing  Steps  Two,  Three,  and  Four  of  the  T&E  process.  It  will 
result  in  Phase  III  technical  and  operational  assessments.  As  a  minimum,  T&E  will 
address  the  issues  in  Figure  3-4. 
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Phase  Ill-Production  &  Deployment  I 

and  fl 

Phase  IV-Operations  &  Support 

For  the  selected  system, 

( 1 )  For  DT&E,  what  are  the  deployed  system’s  demonstrated  capabilities  and 
limitations?  What  are  the  capabilities  and  limitations  of  modifications  and 
upgrades? 

(2)  For  OT&E,  does  the  system  continue  to  be  operationally  suitable  and  effective 
in  operational  use?  Do  proposed  modifications  and  upgrades  increase  the 
suitability  and  effectiveness  of  the  system? 

(3)  Have  performance  objectives/thresholds  versus  advanced  threats  been 
validated?  Does  the  system  meet  these  performance  objectives/thresholds? 

(4)  Is  the  TEMP  current? 


(5)  Should  a  Milestone  IV  be  directed  after  Milestone  III,  what  are  the  Phase  III  T&E 
"exit  criteria?" 


Figure  3-4.  Phase  III  and  Phase  IV  T&E  Questions 


3.9.  Milestone  IV-Major  Modification  Approval.  The  MS  IV  decision  will 
determine  if  the  major  modification  of  the  system  is  warranted.  If  so,  the  DoD 
Acquisition  Phase  to  reenter  will  be  designated  in  the  ADM. 

3.10.  Phase  IV-Operations  and  Support.  This  phase  will  identify  if  the 
fielded  system  is  meeting  the  users  needs  and  if  any  shortcomings  and  deficiencies 
need  to  be  corrected  to  improve  performance.  As  a  minimum  T&E  will  address  the 
questions  in  Figure  3-4. 

4.  Program  Documentation. 

Documentation  requirements  outlined  in  the  DoD  5000  publications  will  remain 
unchanged  except  for  the  TEMP.  The  TEMP  format  will  be  updated  to  include  the 
application  of  the  Process. 
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5.  Summary. 

The  DoD  T&E  process  is  applicable  to  any  system  acquired  by  the  Department 
under  the  provisions  of  the  DoD  5000  series  documents.  This  process  is  defined  in 
more  detail  for  EW  systems  in  -  A  Description  of  the  DoD  Test  and  Evaluation 
Process  for  Electronic  Warfare  Systems,  June  13,  1994. 
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